Adaptive behavioral intrusion detection systems and methods

ABSTRACT

Systems and methods for analyzing historical network traffic and determining which traffic does not belong in a network are disclosed. Intrusion detection is performed over a period of time, looking for behavioral patterns within networks or information systems and generating alerts when these patterns change. The intrusion detection system intelligently forms correlations between disparate sources to find traffic anomalies. Over time, behaviors are predictive, and the intrusion detection system attempts to predict outcomes, becoming proactive instead of just reactive. Intrusions occur throughout whole information systems, including both network infrastructure and application servers. By treating the information system as a whole and performing intrusion detection across it, the chances of detection are increased significantly.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is the U.S. national phase of International ApplicationNo. PCT/US03/09543 filed on Mar. 28, 2003 and published in English asInternational Publication No. WO 03/083660 A1 on Oct. 9, 2003, whichapplication claims the benefit of U.S. Provisional Application No.60/368,628 filed on Mar. 29, 2002, entitled “Adaptive Behavior IntrusionDetection Systems and Methods,” the contents of which are incorporatedby reference herein.

FIELD OF THE INVENTION

The present invention relates generally to methods and systems forproviding security on a communication system and, more particularly, theinvention relates to adaptive behavioral intrusion detection.

BACKGROUND OF THE INVENTION

With the rise of the Internet and the use of computer networks by manybusinesses, network security has become increasingly important. The riseof e-commerce has led to organizations opening their networks to wideraudiences over the Internet in order to stay competitive. Such opennetworks expose the organizations to intrusions—attempts to comprise theconfidentiality, integrity, or availability, or to bypass the securitymechanisms of a computer system or network. Additionally, companiesstoring vast amounts of consumer data need to provide some reasonablemethod for assuring privacy.

Attackers or hackers have continued to alter their attacks and networksubversion methods, and vulnerabilities continue to exist in many areasincluding network misconfiguration, poorly engineered software, userneglect and carelessness, and basic design flaws in protocols andoperating systems. Furthermore, as the sophistication of tools used byhackers has increased, the technical knowledge required to attack anetwork has fallen. Additionally, attacks are often the result ofmalicious insider activity which cannot be prevented by perimeterdefenses.

Intrusion detection is the process of monitoring the events occurring ina computer system or network and analyzing them for signs of intrusion.An intrusion detection system (IDS) is a software product or hardwaredevice that automates the intrusion detection process, and an IDStypically includes three functional components: information sources,analysis, and response. Analysis strategy falls into two basic types:knowledge-based misuse detection and behavioral-based anomaly detection.Behavioral-based detection methods use information about repetitive andusual behavior on the systems they monitor and note events that divergefrom expected usage patterns.

Intrusion detection allows organizations to protect their systems fromthe threats that come with increasing network connectivity and relianceon information systems. Given the level and nature of modem networksecurity threats, IDSs have gained acceptance as a necessary addition toevery organization's security infrastructure. IDSs automatically reviewmassive amounts of network and system data in real time, identifysuspicious activity, provide real-time automated notification tosecurity personnel, guide further investigation, and sometimesautomatically respond to specified attacks. Properly used, an IDS candetect common attacks, attempts to exploit known weaknesses, networkprobes, or critical resource overloads in a reasonably timely manner. Byidentifying successful invalid activity, IDSs can indirectly spotlightnetwork and system vulnerabilities, enabling fixes and fine-tuning.

Comprehensive network security requires multiple layers. An effectiveIDS includes both knowledge-based and behavioral-based components. Mostvendors provide network security products that protect only againstknown or “signature” patterns of attack, and typical behavioral-basedcomponents are limited to single anomaly detection without looking forbehavioral patterns over a longer period of time. Existing productsignore troublesome new behavioral patterns that have yet to be detectedor documented. Hackers often follow certain behavioral patterns thatdouble as calling cards for their personal invasive techniques. Forexample, a hacker may attack all of the hacker's targeted networks by arecognizable and consistent sequence of port access attempts, but thepattern is recognized as odd or alarming only after an attack hasoccurred and a profile for that behavior is documented and publicized.Signature and basic behavioral methods of threat detection areinvaluable, but they fall short as hackers determine new ways to attackor adjust their old behavior to attract less attention.

Many serious intruders perform considerable amounts of probing workwithin a network to learn how it is constructed and understand itsweaknesses prior to a concerted attack. This reconnaissance work iscommonly recorded in automated server and network logs, but largelyremains unnoticed by most network IDSs if the traffic anomaly does notfit the profiles of known or common “signature” hacks. Accordingly,there is a need for a system with adaptive technology that, over time,gathers information on a particular system and establishes a pattern ofnormal traffic. Such a system is able to more intelligently determinewhich network traffic signatures do not fit the normal profiles for theindividual system and alerts an intrusion detection team for furtherinvestigation and appropriate rapid defensive action.

SUMMARY OF THE INVENTION

Systems and methods for analyzing historical network traffic anddetermining which traffic does not belong in a network are disclosed.Intrusion detection is performed over a period of time, looking forbehavioral patterns within networks or information systems andgenerating alerts when these patterns change. Normal traffic behavior iscollected on a continuing basis to establish a baseline for comparisonwith future network traffic. Once a statistically significant sample ofhistorical data has been compiled, a behavioral intrusion detectionagent is activated. The intrusion detection system intelligently formscorrelations between disparate sources to find traffic anomalies. Overtime, behaviors are predictive, and the intrusion detection systemattempts to predict outcomes, becoming proactive instead of justreactive. Intrusions occur throughout whole information systems,including both network infrastructure and application servers. Bytreating the information system as a whole and performing intrusiondetection across it, the chances of detection are increasedsignificantly.

An exemplary embodiment of a method according to the present inventionfor detecting network intrusion attempts associated with network objectson a communications network includes collecting normal traffic behaviorassociated with network objects on the network on a continuing basis toestablish historical data regarding traffic across the network. Networktraffic associated with network objects on the network is monitored todetect anomalies, which are analyzed using the historical data. Alertsare generated identifying possible intrusion attempts based on analysisof the anomalies. The historical data is continually updated based onthe anomalies, the alerts, and network traffic.

In an exemplary embodiment, monitoring network traffic to detectanomalies may include monitoring network traffic for known strings andseries of bytes that indicate signature attacks. Monitoring networktraffic may also include applying a series of rules to identifyanomalous packets and adding the anomalous packets to an anomaly pool.In an exemplary embodiment, analyzing the anomalies using the historicaldata includes analyzing packets in the anomaly pool independently of anyof the series of rules that identified the packet for addition to theanomaly pool. Analyzing the anomalies using the historical data may alsoinclude conducting a threshold analysis to determine whether a datapoint is within threshold values.

In an exemplary embodiment, generating alerts identifying possibleintrusion attempts includes adding alerts to an alert pool and releasingalerts to a console for viewing by an operator. According to anotherexemplary embodiment, generating alerts may include an alert releasesystem, where a first set of alerts are added to an alert pool, internetprotocol addresses associated with each alert in the alert pool areresolved with a name and each alert in the alert pool is renamed with aname recognizable by an operator, a set of rules is applied to selectalerts from the alert pool to be displayed on a console for viewing bythe operator, the rules comprising high level selection parameters thathave been previously defined, and the selected alerts are released byname to the console for viewing by the operator.

Certain exemplary embodiments of methods of the present invention may beperformed across a plurality of networks and with the results compiledin a global database, and historical data may be updated based onresults in the global database.

A computer storage medium storing a computer program, when executed by acomputer-controlled apparatus, may cause the computer-controlledapparatus to perform certain exemplary embodiments of methods accordingto the present invention. Additionally, a computer-controlled apparatusmay be operative for implementing certain exemplary embodiments ofmethods of the present invention.

In an exemplary embodiment of a system according to the presentinvention, an intrusion detection system for detecting network intrusionattempts associated with network objects on a communications networkincludes a sensor connected to the network for monitoring networktraffic associated with network objects on the network. The sensor mayinclude a knowledge-based component for examining network traffic forknown strings and series of bytes that indicate signature attacks and apacket logger for reading packets in network traffic, classifyingpackets by protocols, and creating packages of compressed packets. Aserver connected to the sensor accepts real-time alerts for possiblesignature attacks, and a converter is provided for converting alertsfrom native signature format to a unified format for storage in at leastone relational database. An analysis server receives compressed packetsfrom the sensor at periodic intervals, and the analysis server conductsa behavioral analysis of the data received from the sensor. The at leastone relational database stores raw packet data, behavioral data, andindex data.

Certain exemplary embodiments of systems of the present invention mayinclude a plurality of sensors connected to the network. Also, two ormore virtual private network tunnels connecting the sensor to thenetwork may be provided in certain exemplary embodiments.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is an exemplary environment for operation of systems and methodsaccording to the present invention.

FIG. 2 shows an overview of process flow of an embodiment according tosystems and methods of the present invention.

FIG. 3 shows process flow for an embodiment of resource tracking andrules-based anomaly detection according to systems and methods of thepresent invention.

FIG. 4 shows process flow for an embodiment of anomaly pool analysisaccording to systems and methods of the present invention.

FIG. 5 shows process flow for an embodiment of alert classificationaccording to systems and methods of the present invention.

FIG. 6 shows process flow for an embodiment of statistical generationaccording to systems and methods of the present invention.

FIG. 7 shows process flow for an embodiment of threshold analysisaccording to systems and methods of the present invention.

FIG. 8 shows process flow for an embodiment of alert correlationaccording to systems and methods of the present invention.

FIG. 9 shows process flow for an embodiment of global analysis accordingto systems and methods of the present invention.

FIG. 10 shows process flow for an embodiment of alert release systemaccording to systems and methods of the present invention.

DETAILED DESCRIPTION OF THE INVENTION

In describing embodiments according to systems and methods of thepresent invention, the following terms are used herein in the mannerindicated below:

Agent: An IDS component that gathers system data, monitors systemactivity, and issues alerts upon detection of an intrusion.

Alert: A message sent by an IDS warning of a suspected or actualintrusion and usually calling for some sort of action in response. Alsoreferred to as a notification.

Alert Generation: Addition of an alert to a database to record adetected event. All generated alerts are not necessarily displayed orreleased to an operator of the IDS, but generated alerts remain in thedatabase.

Alert Release: Displaying an alert on a console for viewing by anoperator of the IDS.

Console: An administrative or management component of an IDS. Often agraphical user interface (GUI) through which an operator or usercontrols operations of the IDS and where notification of alerts occurs.

Envelope: A two point data set that contains source and destinationaddresses in a raw packet.

Escalation: An alert is made a high priority.

Generation I statistic: A simple count of packet statistics. Alsoreferred to as first generation statistic.

Generation II statistic: A count of a relationship between multiplegeneration I statistics.

Generation III statistic: A measure of a timed data point of a knownresource (i.e., how much or how often a statistic occurs in a given timeframe). Also referred to as frequency.

Intrusion: A violation of a system security policy by an unauthorizedoutsider or by an otherwise authorized user. A violation may includeimproperly accessing the network, accessing certain systems within thenetwork, accessing certain files, or running certain programs.

Intrusion detection system (IDS): An automated system that can detect asecurity violation on an information system or network.

Normal: This indicates acceptance to a network and is network and sensorspecific, meaning that what may be abnormal on one network may bereadily accepted on another.

Prediction: A calculated value that is a guess of a future data point.This is not to be confused with curve fitting.

Protected Network: A range of addresses a sensor considers internal.Anything with a source outside this range is considered external.Anything inside this range is considered internal.

Protocol: A set of formal rules describing how to transmit data,especially across a network. This is a low level protocol defining bit-and byte-ordering and the transmission, error detection, and correctionof the bit stream. Examples are IP (internet protocol), IPSec, (secureinternet protocol), TCP (transmission control protocol), UDP (userdatagram protocol), ICMP (internet control message protocol), ARP(address resolution protocol), and others which are well known to thoseskilled in the art.

Raw data: Actual packet headers up through layer 4 (64-bytes) stored ina file. Raw data is captured on a sensor and transferred to servers forimport into a database. Once transferred to the servers, the raw data isno longer necessary.

Raw packet: Data pulled from a network packet and stored in a databasein a field by field schema.

Resistance to Change (RTC): The resistance of any tracked data point tobeing changed by an analysis routine (i.e., the data point'spredictability). A data point with a high predictability resists changebecause it has been judged correct many times before. A data point witha low RTC is typically unpredictable because it has a high rate ofchange.

Resource: A usable function of a network device, including servers,protocols, and services.

Rule: A set of selection criteria used to select raw packets.

Score (SCR): A human rating of any tracked data point Similar tostrength but carries a heavier weighting when determining how normal adata point is. A small change in score is the equivalent of a largechange in strength. This value can only be changed manually by anoperator of the IDS.

Sensitivity: A global adjust that may be used to make a sensor more orless sensitive to certain types of alerts, such as resource, signature,threshold, and the like. Sensitivity can be used to tune a sensor tobetter fit a particular network.

Sensor: A device placed on a host network to monitor. A sensor mayperform two functions: knowledge-based intrusion detection and packetlogging of behavioral packages. A sensor collects raw packet data andbehavioral data necessary to perform any analysis. This includes alerts,raw packets, thresholds, frequencies, attackers, and the like. The datacollected remains separate for each sensor, but has the same format andschema.

Server: A network device having an address and transferring data on anetwork.

Service: A high-level protocol that specifies a function that is carriedby a low level protocol. An example is HTTP (hyper text transferprotocol) carried by TCP, DNS (domain name server) carried by UDP, or anEcho Reply ICMP packet.

Strength (STR): An IDS rating of any tracked data point that judges thedata point's normalcy to the network. Typically, low values indicatenormal and high values indicate abnormal.

Strength-Score Value (SSV): A calculated value that indicates theseverity of an alert regarding its danger to the host network. Everyalert has an SSV that indicates the threat assessment of the source ofthe alert This value is calculated using the strength and score of thealert and the alert's source. Typically, the higher the SSV, the higherthe threat.

Threshold: A bracket of a data curve that brackets both above and belowthe data points, indicating the curve's normal values. A thresholdcontains four data points for every data point on the tracked curve. Twopoints are a close estimate of the curve (a thin bracket that is usuallywithin 3-5% of the normal data point) and a second more conservativeestimate that is a wider bracket (usually with 6-10% of the normal datapoint).

Systems and methods according to the present invention provide theability to analyze historical network traffic and determine whichtraffic does not belong in a network. Intrusion detection is performedover a period of time, instead of examining information quickly andnever again seeing the information. Systems and methods of the presentinvention look for behavioral patterns within networks or informationsystems and generate alerts when these patterns change. Normal trafficbehavior is collected on a continuing basis to establish a baseline forcomparison with future network traffic. Once a statistically significantsample of historical data has been compiled, a behavioral intrusiondetection agent is activated. The intrusion detection systemintelligently forms correlations between disparate sources to findtraffic anomalies. Over time, behaviors are predictive, and theintrusion detection system attempts to predict outcomes, becomingproactive instead of just reactive. Intrusions occur throughout wholeinformation systems, including both network infrastructure andapplication servers. By treating the information system as a whole andperforming intrusion detection across it, the chances of detection areincreased significantly.

The analysis of network-wide events is difficult because the data comesfrom very dissimilar sources and analyzing the source data ischallenging. Any deviation in the normal behavior of the informationsystem is a sign of possible intrusion. The data comes from sourcesincluding servers, network sensors, and firewalls, not just a singlesource. Once captured, source data (e.g., raw packet data) is stored andanalyzed over a first specified period of time, and behavioral events(e.g., alerts) are stored and used for analysis for a second, longerspecified period of time. One example is a one month first period oftime and a one year second period of time. It should be understood thateach of the time periods may be varied according to the preferences ofthe IDS users and operators.

Sources of attacks are stored long-term in a historical database alongwith an indicator of their hostility to the host network. This allowsall further anomalies and alerts to be escalated to indicate that theattack was not a single, isolated event but a repeat from a knownattacker. Additional attacks, in turn, raise the indicator leading tofaster escalation, while periods of inactivity from the attacker lowerthe indicator. Attackers are removed from the historic database onlywhen they have decayed to point where they have reached a negativeindicator below a predetermined value. This value is a software setting.Higher values track attackers longer and lower values release them fromthe database sooner.

Additional features of an IDS according to this invention may include:analysis of bit-level packet detail; correlation of packet anomaliesagainst IP-independent attack profiles; easy adaptability that includessignature intrusion detection systems; time based analysis thatcorrelates profiles against packet anomalies; port access attemptscorrelated by both time and sequence; and traffic data stored in andprocessed from a relational database (RDB).

An exemplary environment for operation of systems and methods accordingto the present invention is shown in FIG. 1 and includes a master server(also referred to as master ESP server), at least one real-time server,at least one converter, at least one relational database (RDB), at leastone application and analysis server, a web server, a session server, anda sensor (also referred to as an ESP sensor). Exemplary functionalitiesof each of these components are discussed below.

A master server serves an ESP database 104. Among other things, themaster server keeps track of the job list, including pending, completed,and in-process jobs, and handles cooperative process-locking. The masterserver may also distribute sensors over RDB servers, stores globalconfiguration values, stores sensor master variables (staticconfiguration values for each sensor), and stores user accounts. In theembodiment shown in FIG. 1, a session server 126 acts as the masterserver. It is well understood by those skilled in the art that anyappropriate server may be the master server. Servers such Oracle orMySQL servers or other standard hardware running Linux/Unix basedoperating systems may be used for the master server, as well understoodby those skilled in the art.

As shown in FIG. 1, real-time servers 106 and 108 accept sensorconnections for real-time alerts. Real-time servers 106 and 108 maintainthe sensor real-time tunnel (network connectivity), store real-time datain native format (native to signature system: Dragon, Snort, etc.),store logged data from events (actual data capture at time of event),rebuild data sessions for application servers, and may also act asconverters. Real time servers are standard servers, well known to thoseskilled in the art, running Linux/Unix based operating systems.Typically, they may include a dual-processor based system with a RAIDarray disk unit and a network file system, such as CODA, CIFS, or NFS,if the disk storage is not already network-based, all of which is wellunderstood by those skilled in the art.

A converter 110 reads real-time alerts from native formatted files,connects to a sensor's RDB (such as RDBs 112, 114, and 116), convertsalerts from native signature format to unified format, and may combinemultiple sensors into a single ESP sensor. Converter 110 is standardhardware, well known to those skilled in the art, running Linux/Unixbased operating systems that is able to mount network shared filesystems and appropriate client software to connect to the RDBs.Relational databases 112, 114, and 116 (RDBS) store raw packet data,store behavioral data, and index data. RDBs 112, 114, and 116 sendserver data to web servers (such as web server 124), application servers(such as application server 120), and analysis servers (such as analysisserver 122). RDBs 112, 114, and 116 accept data from converter 110.Additional storage, such as storage 118, may also be provided forstorage from RDBs 112, 114, and 116 or database 104. RDBs 112, 114, and116 may run on standard SQL-based database servers, such as those fromOracle or MySQL, which is well understood by those skilled in the art.The RDBs may typically have 1 GB of RAM per processor, and someconfigurations may include a storage area network (SAN) or networkattached storage (NAS) for central database storage.

An application server or servers 120 stores the system's backend userinterface code, serves application data to web servers (such as webserver 124) and other user interface systems. Application server 120 mayserve any sensor. An analysis server or servers 122 looks for pendingjobs, locks jobs, marks jobs in-process, performs analysis jobs, storesresults (update behavioral profile), marks jobs processed, and unlocksjobs. Application server 120 and analysis server 122 may be combined ina single server. Application and analysis servers include standardhardware running a Linux/Unix based operating system with appropriateclient software to connect to the RDB servers, as well understood bythose skilled in the art. Open SSH, which is well understood by thoseskilled in the art, may be used if encrypted communications are requiredfrom the application and analysis servers to any web servers.

A web server 124 processes web requests, calls application serverapplets to produce application data, and handles user interfaces. Webserver 124 may not access sensor data. A web server is standardhardware, well known to those skilled in the art and typically runs on aLinux/Unix based operating system with appropriate client software toconnect to RDB servers. Session server 126 handles web server sessionkeys, stores session data, deletes old sessions (stale), and handlesaccounts. As noted above, session server 126 may, in some instances, bethe master server. Standard server hardware may be used for the sessionserver, such some of the servers described above, as is well understoodby those skilled in the art A separate relational database 128 connectedto web server 126 may be provided.

A sensor (or ESP sensor) 102 includes three separate subsystems designedto perform three functions: signature checking, data capture, andsecurity. Each sensor is a bastion host containing an internal firewall,a knowledge-based sensor, and a packet logger. The knowledge-basedsensor is designed to handle signature detection only, and leaves theanomaly detection to the IDS analysis servers. The packet logger hascomplex filtering capabilities that can be used to capture only thedesired data, but this is rarely employed. The more data the systemcaptures, the better the analysis. Typically, sensor 102 includes asingle or dual processors (for larger networks) and two networkinterface cards. Standard hardware runs Linux 2.4+ with the Linux socketfilter, sock packet and ipchains features enabled. A virtual privatenetwork tunnel or other transport tunnel is necessary to move data fromthe sensor to a server. As noted above, OpenSSH may be used to transferbehavioral packages to conversion and import servers.

Sensors are designed as bastion hosts because they are employed inperimeters in front of any security devices. Typically, the sensorsoperate with a read-only wire for the sensing interface, but a secondnetwork interface handles communications with the console, and as asecurity device, this interface must be well protected. The internalfirewall is used to protect the communications interface and to protectthe sensor for denial of service attacks using traffic shapingtechniques. All packets on the communications interface are fullydefragmented before being passed to the remainder of the IDS.

As noted, sensor 102 includes a packet logger. On very high-speednetworks, any analysis on the sensor hinders performance of the sensorand can lead to dropped packets. For these reasons, the behavioralanalysis occurs offline. The packet logger creates “packages” ofcompressed and encrypted traffic that is sent back to the analysisstation at specified intervals, for example 30-minute or 60-minuteintervals. Each package is unencrypted, decompressed, and fed into thebehavioral analysis portion of the IDS, as further described below.

The packet logger reads packets from the network interface, classifiesthe packets by protocol, compresses the packets, and writes them to adisk using a double-buffered, threaded process. The analysis reliesmainly on traffic patterns of a network, so the data is not necessary.Though there are reasons to log the data, it is not essential forbehavioral analysis. On very high-speed networks, the sensor can besplit into two parts, with one performing signature checking with aknowledge-based sensor, and the other performing packet logging. Thisallows for maximum speed for each data source and packet logging tospeeds of 180+Mbits.

A firewall 130 hides the remainder of the IDS from host network 134 andprotects sensor 102. Any standard state-aware firewall well known tothose skilled in the art is acceptable for use. Firewall 130 also splitstraffic to the various servers, as shown in FIG. 1. Sensor 102 initiatesvirtual private network (VPN) tunnels 132 to firewall 130. VPN tunnels132 include a long term encrypted tunnel for real time alerts and ashort term tunnel for transactions that connects when data needs to betransferred and disconnects when data transfer is complete. In oneembodiment, data is transferred using two tunnels and maintenance isperformed in a non-real-time mode. In other words, the IDDS transfersknowledge-based sensor alerts through a knowledge-based sensor tunneland all sensors share a single tunnel for raw alert storage on thereal-time server. In another embodiment, an e-tunnel allows for the useof any signature-based product with a behavioral system. Additionally,the e-tunnel allows for transfer of multiple growing files to theserver, handles multiple input and output streams, creates and maintainsa VPN tunnel (robust encrypted network connection), performs systemmaintenance remotely and securely, reports on system status (performancemonitoring), and provides each sensor with its own data store on theserver.

Sensors typically communicate back to the remainder of the IIDS using asecured method. Network sensors minimally employ a 640-bit encrypted TCPtunnel. The communications key is changed periodically, e.g., hourly.Authentication occurs using, for example, a 768-bit certificate on bothsides of the tunnel. The firewall only allows connections between theregistered analysis station and the sensor, the tunnel applicationverifies the host Internet address, and both Internet addresses are usedin key generation and authentication exchange. Remote sensoradministration is performed through the same 640-bit channel. Allcommunications to the sensor, including performance monitoring, arefully encrypted and authenticated through the same mechanism.

FIG. 2 provides an overview of process flow of an embodiment accordingto systems and methods of the present invention. In an exemplary process200, knowledge based intrusion detection including sending signaturealerts in real-time, block 250, is included. The function ofknowledge-based intrusion detection is to perform real-timevulnerability checking of packets that are passed through the firewall.The knowledge-based portion of the sensor is charged with datainspection and alert generation from signature checking. Similar tovirus checking, a sensor looks in network packets for known strings andseries of bytes that indicate attacks. Alerts appear on the console inreal-time. The alerts are automatically prioritized by informationcontained about the source address in the behavioral database, whereasmost signature-based processes prioritize alerts by type (alert name)only. The signature-based alerts are escalated from their defaultpriority as necessary based on the source's strength, score, and thesignature sensitivity setting for the sensor.

Behavioral analysis includes block 300 through block 800. In blocks 300through 800, behavioral analysis is performed for an individual hostinformation system (a collection of related sensors). Finally, globalanalysis occurs at block 900 and includes behavioral intrusion detectionover sensors across multiple information systems in order to help findhackers across the Internet. This can identify scanning sources andcoordinated attacks.

Behavioral analysis includes adaptive rules-based profiling usingmodel-based reasoning. Behavioral patterns are abstracted by describingthem as sequences and thresholds stored as historic rules. Currentbehavioral patterns are then checked against the historically predictedpatterns and deviations are noted. Some deviations can be classified atthe abstract level, while others require the IDS to find the associatedsource data and perform rules-based source analysis. Behavioral analysisuses information in the RDBs with the goal of detecting all intrusions.The sensor is connected to the remainder of the IDS and transfers thelogged packets to the analysis server. Each packet is decompressed andimported into the RDB for the particular sensor, and analysis thenbegins. Since analysis is rule-based, once the data is ready, processingcan be split across multiple servers allowing analysis to occur onclusters.

As shown in FIG. 2, behavioral analysis includes numerous steps.Resource tracking, block 300, is a process that discovers networkinformation by examining the host network traffic. The analysis serverlooks for servers, protocols, and services and records them in thedatabase. After being recorded, these services, protocols, etc. aretracked in the future and statistics about them are gathered. Thisprocess is continuous, so new resources are discovered and old resourcesare deleted, generating alerts in either case. By listing this data,systems, protocols, and services in use at any given time are identifiedand a measure of how important they are to the host network is provided.

Rules-based anomaly detection, block 350, includes using a series ofrules to find anomalies on the network. At this point, anomalies areindividual packets that match selection rules that define such things asillegal packets, unusual packets, and normal packets that are not sentto valid network resources (e.g., http requests to a server that doesnot service http). These packets are collected and added to the databasein a protocol independent area called the anomaly pool. There are aseries of normal rules that then subtract known normal packets—thisaccounts for anomalies that are accepted as normal in the particularhost network.

Also included in stage 2 is anomaly pool analysis, block 400. Once ananomaly pool is created, the packets in the pool are examinedindependently of the selection process. This may be referred to as“blind” analysis because this analysis is carried out without the serverhaving any knowledge of why the packet was added to the anomaly pool.For instance, if a rule added packets because the packets have anunusually low time-to-live (TTL), it would be simple to generate analert that says “LOW-TTL.” However, this is biased, and presupposes aLOW-TTL and nothing else about the packet. By examining the packetblindly without knowing why the packet was added to the anomaly pool,the blind analysis attempts to determine why the packet was added. Thisleads to relationships between packets and anomalies that indicatebase-cause issues and is comparable to a pathologist finding the originof a disease as opposed to just treating the symptoms. The blindanalysis aids in finding new attacks because when the entire anomalypool cannot be accounted for, human operators of the IDS are notified toconduct further analysis to determine why this is the case. The resultis generation I anomaly alerts that are classified, block 500.

At block 600, statistics are generated. Generation I statistics aregenerated, which involves a simple counting of various aspects ofprotocols, such as number of packets, incoming packets, outgoingpackets, categorizing packets into various sizes, etc. These statisticsare referred to as generation I because they varying and are lesspredictable. Generation I statistics do not show relationships oftraffic flow, but they provide a volume baseline. Next, generation IIstatistics are created. This process, using protocol and service models,relates statistics to create more predictable values that show trafficflow as opposed to volumes. For instance, a comparison of inbound pingrequests to outbound ping replies may be made. Generation II statisticsillustrate a relationship between packets that creates a predictabletraffic flow. The difference between these values as a percentage of thepackets is almost always a predictable value on networks. Generation IIstatistics are stored in the database. Generation III statistics (alsocalled frequencies) are calculated as well, counting resource usagevalues and envelope pairings that track which resources are used bywhich systems.

Threshold analysis is performed at block 700. Threshold values arestored based on historical data. Threshold violations tend to show largeproblems, but determining metric overflow and underflow values isdifficult. For this reason, sequences of values are often aggregatedtogether using the model-based reasoning system so that thresholdindicators are dependent on more than one possible value. If theabstracted threshold fails, the source data is examined and comparedagainst the model rules once again. Since data is compared in theabstract layer to historical data, historical source data is necessaryfor full analysis. For each statistic, both generation I and generationII, each value is checked against the most recent prediction. If itfails, an alert is generated. New predictions for the next time interval(e.g., an hour) are calculated and stored. Generation II alerts areweighted heavier then generation II when creating alert prioritizations.Frequencies are checked by threshold predictions. Any remainingviolations are added to the alert pool and submitted for adaptiveclassification.

After threshold analysis is complete, alert correlation occurs, block800. Alerts are correlated by type using statistical analysis of thetypes of alerts received. The correlation measures the number of alertsof various types and relates increases and decreases of alerts and theirrelationships to the percentage of the whole. The correlation is thenperformed for all alerts on the specific network. Related sensors areanalyzed and alert relationships by source of alert (attacker) arenoted. Attackers are recorded and tracked in each individual sensor.Upon completion of alert correlation, behavioral analysis for aparticular sensor is complete.

Global analysis, block 900, includes analysis of all available sensorsas if they were on one large network. For example, if an IDS providerservices one hundred networks or information systems, the globalanalysis may be performed for all sensors on these one hundred networks.Global analysis begins with the grouping of all alerts from all sensorsinto a common alert pool. Where alerts are generated becomes irrelevant,only the alert types and sources are significant. The common alert poolis then examined by alert type and by source, similar to the stepsperformed in the behavioral analysis for each sensor. This yields newprioritization values of both attackers and alert types. This newinformation may then be sent to the sensors to help prioritize alerts ona per sensor basis, block 950.

More detailed information about certain exemplary embodiments ofprocesses in blocks 300-950 of FIG. 2 is shown in FIGS. 3-9 anddescribed below.

Referring now to FIG. 3, process flow for an embodiment of resourcetracking 300 and an embodiment of rules-based anomaly detection 350according to systems and methods of the present invention are shown.Initially, data is split by protocol and date in an RDB server, block290. Periodically (e.g., hourly, bi-hourly), packets collected on asensor are sent to the RDB server. These packets are split by protocol(e.g., ICMP, IP, TCP, and UDP) and inserted into the database. A filtermay be applied at the time of import to selectively insert only certainpackets, which is helpful in specific situations where it is notdesirable to send certain packets to the database. At block 295, importpackets are indexed in the relational database.

Resource tracking begins at block 305. Resource tracking examines theraw data and finds resources on the host network. Resources are found,block 310, using protocol models. Multiple algorithms pass through theraw data finding resources. Some are simple, such as looking in theprotected network for internet addresses that indicate servers. Othersare more complex and require examining data flow to find the protocolsand services. Resource activity is verified by finding two-wayconversations, not simply looking for outbound traffic. The conversationis verified by finding evidence of traffic moving to and from systemsthat appear to be servers. For some protocols, this is a statisticalanalysis, and for others it is a search algorithm.

At block 315, if the resource is a new resource (i.e., the IDS has notpreviously “found” this resource), an alert is added. The resource isadded to the resource database with its STR, SCR, and RTC set to zero.If the resource is already listed in the database, block 320, its STRincreases. If the resource was found within a predetermined time (e.g.,the last hour), the RTC is incremented. Once all new and old resourcesare analyzed in blocks 315 and 320, the resources are pooled together,322, and resources that are in the database but were not modified loseSTR, block 325. The decrease in STR is inversely related to the RTC.Resources with high RTC lose little STR, while resources of low RTCdecrease rapidly. At block 330, client resource normalizing rules areapplied. Because some systems or certain protocols or services arerarely used, a mechanism to allow overrides is included. To accomplishthis, a set of static normalizing rules define resources that do notchange and/or resources that are not to be added.

Rules-based anomaly detection 350 begins at block 352 with examinationof new packets. All raw packets are examined for anomalies. Two types ofanomalies are searched for: anomalous packets and normal packets thatshould not occur on the network. These packets are added to the databasein a table, known as the anomaly pool, that is protocol independent. Aresource list output from resource tracking 300 is added to the newpackets, 354, and rules of rules-based anomaly detection are applied, atblocks 356-368, to find anomalous packets. The first rule applied, block356, is adding all packets that match static selection rules. Staticselection rules add to the anomaly pool anything that matches. These arenormally added to track traffic that is completely normal, but is beingtracked for some external reason, such as alerting all traffic from aparticular source. At block 358, all protocol violations are added.Protocol violations are problems with packet data that are not allowedby the protocol specification or are not specified by the RTC. Thesepackets are not normal to a network because they violate the protocolrules, but can still be routed on a network.

Packet port violations are added at block 360. Packet violations areusually protocol violations, but are found by the IDS in a differentmanner. Unlike protocol violations, packet port violations are packetsdropped off the network by hosts upon delivery. The packets are somalformed they are dropped completely, such as a layer specifying thatlayer 4 is TCP when the packet does not conform to TCP. At block 362,packets from envelope pairings are added. These are packets that couldnot have been generated or delivered to the host network and are veryoften the result of spoofing activity or an envelope specifying the samesource and destination. Type code anomalies are also added, block 364.All ICMP packets have a type and a code that specifies what type ofmessage the packet is carrying. Type code rules find illegal or strangetype code combinations.

At block 366, type of service (TOS) anomalies are added to the pool. TheTOS field is a part of the IP protocol. Only certain values can be inthe TOS field and any value outside of these specified values issuspect. At block 368, TCP session indicator violations are added. TCPstate indicators are bits within protocols that are used to indicate orstore the information necessary to the protocol during normal use. Manyprotocol specifications only specify what the indicators are used for,but not how to deal with conflicting indicators or unused bits in theindicator space.

Because broadcast traffic has no specific destination, broadcast trafficis examined separately and packets are added, as necessary, based onprotocol models, at block 370. For instance, TCP to broadcast addressesis not supported and should not be found on a network.

While blocks 356-370 relate to adding anomalous packets to the anomalypool, block 372 subtracts normal packets from the pool. To find normalpackets that meet all specifications and are free of violations but thatare sent to or originating from the wrong resources, all raw packets areconsidered and those that originate properly and have come from properdestinations, as compared to the resource list from resource tracking,are removed. This removes all expected traffic from the raw packets. Theremaining packets are added to the anomaly pool for analysis. At block374, synchronization is performed to ensure that all add (blocks356-370) and subtract (block 372) rules are complete before continuing.

Once again, some traffic added to the anomaly pool may be considerednormal to a particular network, so a set of static deletion rulesremoves all matching traffic from the anomaly pool, block 376. At block378, normal rules are applied. Normal rules are code-based, specificalgorithms that handle special known situations, such as protocol stacksthat generate certain anomalies and the like. The anomaly pool iscomplete, block 380, and all packets are marked “unclassified.”

FIG. 4 shows process flow for an embodiment of anomaly pool analysisaccording to systems and methods of the present invention. As anomalypool or “blind” analysis 400 begins, the anomaly pool is full ofsuspicious traffic to be analyzed and classified. Without knowledge ofwhy the raw packets are in the pool, routines now try to classify eachanomaly or set of anomalies. The first set of routines are anomalies bypattern, block 405. This is a series of code-based algorithms thatclassify packets based on known patterns. These algorithms may bedeveloped using previous unsuccessful attempts to classify packets usingthe steps at blocks 405-430. Any packet or set of packets that matchesthese patterns is marked as “classified” and appropriate alertsgenerated, block 440. A standard add alerts routine is called to createalerts with adjusted SSV for display on the console. These alerts areavailable for viewing.

At block 410, analysis continues with the examination of sequences.Sequences are events in order of time. The examination of sequenceslooks at anomaly pool-based known sequences and is a traffic flowanalysis based on known protocol and service attack models. This is adata driven list of events to look for in the proper sequence forward intime. If any known sequences are found, those classifications areapplied, block 415, the alerts are marked classified, and alerts areadded to the console, block 440. At block 420, a scan detection routineis run. This statistical routine analyzes the distribution ofdestination ports over source addresses. A source hitting a high numberof ports is port scanning, and the number of destinations determineswhether a single host or multiple hosts are scanned. If scans are found,alerts are added to the console, block 440.

Sweep detection, block 425, uses a statistical routine to find sourcesattempting to access a single service on multiple servers, whichindicates a sweep. If sweeps are found, alerts are added to the console,block 440, and the packets are marked classified. Any remaining alertsin the pool are unclassified and are classified by human analysts, block430. These packets are logged and submitted for review. Newclassifications may be entered as sequences or anomaly patterns. Allunclassified anomalies are scanned for sequences that are entered intothe sequence list, block 435. This list can be reapplied to furtherclassify remaining anomalies.

FIG. 5 shows process flow for an embodiment of alert classificationaccording to systems and methods of the present invention. Alertclassification 500 begins with examination of signature alerts at block505. All alerts generated for a specified time period (e.g., an hour,two hours, etc.) are analyzed for further correlations in an attempt togenerate more useful alerts. Alerts are grouped by time and the numberof alerts are counted for high activity periods, block 510. Extremelyhigh numbers indicate further analysis is necessary. Active periods alsoindicate times a hacker may choose to work, which is often a detectablepattern that can be used to find hackers when they switch addresses orattack origins.

Alerts are grouped by source, block 515, over time using historicaldata. This allows the IDS to find slow activity where the attacker isperforming actions slowly in an attempt to stay hidden by not generatingalerts quickly. Such things as slow port scans, slow vulnerabilityprobes, and slow attacks may be found. The attackers' database from theadd alerts routine already tracks individual sources, so this analysisfocuses on slow activity. At block 520, packet flow correlation isexamined, also known as backscatter. Here, protocol models are used toexamine packet flow. Communication sessions may be examined and comparedto normal models to find deviations. Spoofed traffic on the hostnetwork, as well as the residual traffic generated when the hostnetwork's addresses are externally spoofed, may be found. As shown inFIG. 5, historical look-ups, 530, may performed for assistance asnecessary among blocks 505-520 by the IDS.

All historical alerts are grouped by source and counted to determine thedegree of activity for any source using any high frequency alerts, block525. Once the processes in blocks 510-525 are complete, alerts aregrouped by source and appropriate STR changes are calculated for eachalert or time pattern, block 535. Alert patterns warrant high SSVincreases, while time patterns results in only small adjustments. STRchanges are applied to the attack sources, block 540. This increases thestrength of all sources generating like alerts and results in higherSSVs for sources using like alert patterns or like times.

At block 545, technique discriminators are applied to STR. A statictable contains a list of alerts and a number representing the skilllevel required to perform the type of attack that generates that alert.This is representative of the intelligence level of the hacker or, atthe very least, the hacker's ability to know when to apply theappropriate hacking tool. Applying this number as a modifier to STRallows SSVs to be affected by attacker skill in an attempt to highlightstealthy or intelligent hackers that are above average. This is known asa “technique discriminator.” The actual value of the techniquediscriminator is static and assigned by human analysts. The value may beused globally across multiple sensors and multiple protected networks.

A final step of alert classification is applying an RTC bias, block 550,which may be applied to any attack source. Most attack sources have noRTC assigned or tracked, but in some cases, it may be convenient toapply an RTC bias to certain attack sources to prevent a plannedone-time event, such as a vulnerability scan, to affect attackerprofiles. It is typically easier to adjust the score to a low numberbefore the event and reset the strength and score when the event hasoccurred. Because this is effective, RTC values are not stored for everyattacker.

After alert classification 500, two remaining processes may be performedin parallel. Statistical generation and threshold analysis, which areshown and described in FIGS. 6 and 7, and adjusting of attacker values.An exemplary process of adjusting attacker values is briefly describedbelow.

All origin-based alerts (i.e., alerts with a known origin) are generatedand have calculated SSVs in the alert database (and have been shown onthe console). Attackers that have ceased attacking slowly return tonon-threatening status based on the maximum SSV calculated. Since thisnumber is a combination of strength and score, reducing strength to zerowill also zero the threat profile. Attackers slowly decay until they areremoved and this is accomplished by reducing STR for any specified timeinterval (e.g., an hour or three hours) in which the attack source isnot the origin of an alert. The last time an attacker was adjusted isstored, and, for any specified time interval in which an attacker is notthe source of an alert, the STR is reduced by an amount that iscalculated from the period the attacker has not generated an alert andthe attacker's RTC (if assigned). In this manner, attackers that quitgenerating alerts slowly reduce in threat and eventually return to zero.It should be noted that this process is slow and SSVs continue to behigh for some time (how long is based on the SCR) after the attacker hasceased the current attack.

Referring now to FIG. 6, process flow for an exemplary embodiment ofstatistical generation 600 is shown. Generation I statistics arecomputed by simple counting of raw packets, block 605. Block 645 is ageneric descriptor indicating that data is posted to the relationaldatabase. Generation I statistics are less predictable and thereforeharder to track. Generation I statistics include counting frequencies oftypes of packets, such as the number of packets in each protocol, thenumber of incoming packets, the number of outgoing packets, the numberof fragmented packets, and so on. Queries are made on the raw packets todetermine these counts for a specified time interval (e.g., a half-hour,an hour). Some data flow analysis is necessary to determine somegeneration I statistics because certain protocols, such as TCP, requiremultiple packets to determine if sessions are valid or just attempted.Many generation I statistics are gathered and stored, resulting incontinuous data curves that have periodic (e.g., hourly, daily) datapoints. The statistics to be gathered are based on the protocolsanalyzed. For generation I statistics, because the data points do nothave to be predictable, gathering as many statistics as possible is theprimary goal. If predictable values are gathered, it is advantageous,but values that are predictable for one network may not be predictablefor another. Generation I statistics are stored in a separate table.

At block 610, generation II statistics are created from generation Istatistics by finding relationships between data points. This istypically an operator (human) process, but is based on protocol andservice data flow models. A standard set of generation II statistics maybe generated by using the protocol models to find ratios andrelationships. For example, comparing TCP packets with the SYN(synchronize/start) flag turned on in one direction versus TCP packetswith the SYN-ACK (synchronize/acknowledge) flags turned on in the otherdirection. Generation II statistics use two or more generation Istatistics to create more predictable values of related information bycalculating and comparing ratios and differences over time on thenetwork. Generation II statistics are stored in the thresholds table andcalculated during threshold analysis (see FIG. 7 and accompanyingdescription below). The step shown at block 610 is the recording andstoring of values to use as generation II statistics using analysistools that assist in finding useful, predictable generation IIstatistics.

At block 615, generation III statistics are recorded and stored.Generation III statistics calculate events over time. These are calledfrequencies because they track the occurrence of events over time. Threemain types of generation in statistics are calculated: envelopepairings, service usage by resource, and external service usage. Overtime, generation II statistics produce highly predictable curves if theyare adapted as the network changes.

Envelope pairings are calculated by counting the number of times twosystems exchange information. All envelopes are counted and grouped,block 620, by unique pairs of servers. For example, src to dst and dstto src are grouped the same. The number of unique conversations isrecorded as a statistic. This shows which servers talk to which servers,and records the time of the events. At block 625, usage counts ofservice resources are made by examining the resource table and countingservice usage per server. These values are recorded as a statistic andshow the number of times each given resource is used on the protectednetwork, including recording the time of the events. Service usage iscalculated in an outbound direction, block 630, to profile what externalservices the protected network uses. This provides the external servicesused and number of times the services are used and records the time ofthe events.

All counts calculated in blocks 620-630 may be adjusted by a staticdiscriminator, block 635, or removed entirely as a tuning mechanism forthe sensor. Such adjustment or removal may be performed to remove suchthings as traffic generated from ad hoc backups that may distort thestatistic in the historic database (because historic numbers are used tocontinuously adjust the predictions, such large changes may cause animbalance for a lengthy period of time if not removed). At block 640,additional RTC changes are performed. Once again, this is a tuningmechanism used to force a static value onto an RTC to stop an ad hocevent from distorting values in the future. All statistical values arenow calculated and may be compared as thresholds to the last timeinterval's predictions, as shown in FIG. 7.

FIG. 7 shows process flow for an embodiment of threshold analysis 700according to systems and methods of the present invention. Thresholdanalysis includes a curve prediction algorithm that tracks data pointsover time in order to make predictions of future values. Thresholdanalysis may be used to track any statistical value, such as number ofpackets of a given protocol, or data points, such as the frequency of aparticular alert from a given source. Statistical value refers to thosevalues that come from counting events, for example, but not limited to,number of packets, number of UDP packets, number of failed TCPconnections where an RST flag is detected, and others. The statisticalvalues come directly from raw packet data. A data point originates froma pattern the IDS has detected, either automatically or by request froman operator of the IDS. Data points do not come directly from raw packetdata, but rather are calculated from correlated events and represent anyrepeated series of events that occur in a predictable fashion (i.e., apattern). Some examples include, but are not limited to, the number oftimes a particular server on the Internet generates a specific alert,the distinct number of times a worm visits the monitored network, andthe average frequency of web-based attacks for a given server. Bothstatistical values and data points are treated in the same manner bythreshold analysis.

Referring now to FIG. 7, this process iterates once for each threshold.A prediction is made for each collection interval. This prediction isthen checked against the actual data, which may generate alerts. Thethreshold is then stored for future use along with the actual data to beused to calculate and adjust future thresholds. The last prediction madeis stored in the threshold table (marked “prediction”) along with allprevious predictions (marked “archival”). In this manner, the predicteddata curves are constantly changed as the network changes. Allthresholds can be calculated, for example, continuously (all historicaldata points), by time ranges (e.g., from 18:00 to 07:00), or asindividual hours (e.g., from 06:00 to 07:00) and each of these can bepredicted. For each date-time range, there is a separate prediction.This allows for normal behaviors, such as outbound web usage increasingfrom 08:00 to 18:00 on weekdays (a very normal behavior if userworkstations are in the protected network), to be taken into account.

Alert correlation begins against the predicted threshold (calculated atthe last analysis run) in block 702. The current statistic or data pointbeing analyzed is compared to the last prediction, block 704. Theprediction consists of four values: the actual data point, the predicteddata point, the minor offset, and the major offset. The data value iscalculated using the new data received in the current data package(after all the statistics have been generated). The prediction of thecurrent statistic/data point being analyzed is retrieved from thedatabase along with the major and minor offsets. Adding the minor offsetto the predicted value creates the upper limit and subtracting the minoroffset from the predicted value creates the lower limit. This is doneagain with the major offset to create the major bracket. These valuescreate two brackets above and below the prediction that provide a bufferfor the prediction to succeed. The higher the predictability of thegiven statistic/data point, the narrower the minor and major offsets andthe smaller the brackets.

A determination is made as to whether the current statistic or datapoint is outside the major bracket, block 706. If the current statisticis outside the major bracket, this indicates a major threshold overflow,if greater than the upper major bracket, or major threshold underflow,if less than the lower major bracket. If the current statistic isoutside the major bracket, an alert is added to the console, block 708,indicating the major threshold violation, but because the source of theproblem is unknown, there are no changes made to the attacker databaseand the envelope is null. If the major bracket passes, the minor bracketis checked in the same way, block 710. Otherwise, the minor bracketcheck is skipped. If the current statistic is outside the minor bracket,an alert is added to the console, block 712, reporting the minorthreshold violation. Once again, the source is unknown so the envelopeis null.

If either a major or minor threshold violation occurs and an alert isadded to the console, the next prediction is calculated at block 726,which is further described below. If there are no major or minorthreshold violations, meaning the last prediction made is correct, theRTC is increased at block 716. This makes the prediction value moreresistive to change as predictions become more accurate. An accurateprediction over a long period of time has a high RTC and refuses tochange the value at all until the RTC is lowered sufficiently (in block748). In block 718, the STR is increased on a successful prediction by apercentage of the short-term predictability. For example, in a systemwith analysis every hour, the STR can increase more than twenty-fourpoints in a day, if the number of successful predictions in a row ishigh. A predictability value is increased at block 720. If the RTC issufficiently low (a static value), the actual value from the statisticsmay be archived, block 722.

At block 724, the trend of the prediction is calculated. This is not thetrend of the data curve but the trend of the prediction, which is calledthe predictability and measures the ability of the system to track aspecific data curve. This is tracked using two values, one indicatinghow many consecutive times the prediction has been correct or incorrect,measuring the ability of the current settings to predict the data, andthe percentage the prediction is correct overall (i.e., the long-termpredictability of the data). The combination of these two values is thepredictability of the data. If particular data points never stabilize,they can be dropped. If they were previously predictable, an analysisroutine may be run to adjust the number of zones, the target percentagesfor the three tiers, the time ranges, and the sample size of historicdata (these parameters will be better understood by referring to thediscussion below beginning with block 726). Each time a single variableis changed, the new parameters are retroactively applied, and theresults data is scored based on its accuracy with the actual data. Thehighest scoring set of parameters can be instated on the nextprediction. This external routine is typically only run when short-termpredictability is unstable for a data curve with a high long-termpredictability.

Beginning at block 726, a new prediction is calculated following eitherblock 708, 712, or 724, as shown in FIG. 7. At block 726, a distributionover time is calculated as follows. First, a range is found providingthe values to use. Multiple predictions may be made for a single datacurve breaking it into ranges of time. The minor and maximum historicvalues are calculated, giving the upper and lower extents of the value.By subtracting the lower extent from the upper extent, the result is therange over which the data curve varies. This range is divided into anequal number of zones starting at the lower extent and extending to theupper extent. The number of zones may vary, but is typically at least 10and less than 20. The distribution of the actual previous samples, fromthe historic data, over the zones is calculated, and a target percentageis selected. The target value comes from the curve's predictability, oris 10%, for example, if there is no historic data. The bracket isadjusted based on distribution by dropping upper and lower zones that donot contain more than the target percentage of data, and the zones arerecalculated with the new extents. This continues until the target isachieved or the process fails.

At block 728, a tier one bracket is calculated. The tier one bracket iswide with a target of 10%. If the data conforms to the tier one bracket,block 730, (that is, 90% of the curve hits a single zone), theprediction is set for the median of the final extents. If tier one isachieved at the 10% target, then the new prediction is added to thedatabase and the old prediction is archived, and a tier two bracket iscalculated, block 732. If the data does not conform to the tier onebracket, then the prediction fails, block 742. The predictability trendvalues are adjusted downward, block 744. The number of consecutivesuccessful predictions in a row is set to −1 to indicate the trend isone failed prediction. Long term predictability is recalculated. The STRis decreased, block 746, by a percentage of the short-termpredictability. The RTC is decreased, block 748, but only by a smallamount. Once all thresholds are checked and adapted to current networkbehaviors, alert correlation begins (FIG. 8).

If a tier two bracket is calculated, block 732 the tier two bracket isslightly smaller, for example, with a target of 6%. If the data conformsto the tier two bracket (that is, 94% of the curve hits a single zone),block 734, the prediction is set for the median of the final extents. Iftier two can be achieved at the 6% target, the new prediction is addedto the database and the old prediction is archived, and a tier threebracket is calculated, block 736. At block 740, the tier one predictionis used if the tier two comparison fails (i.e., the data was notpredictable enough to hit the 6% target) at block 734.

The tier three bracket is very narrow and is targeted at, for example,3%. A tier three bracket is calculated, block 736. If the data conformsto the tier three bracket (that is, 97% of the curve hits a singlezone), block 738, the prediction is set for the median of the finalextents. If tier three can be achieved at the 3% target, then the newprediction is added to the database and old prediction is archived. Atblock 740, the tier two prediction is used if the tier three comparisonfails (i.e., the data was not predictable enough to hit the 3% target)at block 738. The tier three prediction is used at block 740 is the tierthree comparison was successful. As the prediction is compared to eachof the brackets, the prediction data value or point is the same, onlythe allowable offset changes. The prediction is stored as the value, theoffsets, the target value, the zone size, number of zones, and so on(i.e., everything necessary to calculate the next prediction). Once allthresholds are checked and adapted to current network behaviors, alertcorrelation begins (FIG. 8).

FIG. 8 shows process flow for an embodiment of alert correlation 800according to systems and methods of the present invention. At this stageof behavioral analysis, all raw data has been analyzed. Alertcorrelation examines all alerts for the sensor to find recognizablepatterns that may be used to further classify alerts. Once correlationhas been performed for the sensor, correlation is performed across allsensors on the protected network. Alerts are grouped by source, block805, so they may be easily examined. At block 810, if known sequences ofalerts are found, additional alerts are generated, block 845, to furtherclassify the attack. For example, eleven IIS:CMD.EXE alerts, threeIIS:UNICODE2 alerts, and two IIS:UNICODE2 alerts all from the samesource in a specific order is a Nimda worm attack. The additional alertgenerated is WORM:NIMDA.

The sources are grouped by alert type, block 815. This shows a count ofdistinct sources for each type of alert. At block 820, it is determinedwhether there is a high number of distinct sources. The higher thedistinct count of sources for each alert, the more active the event(associated with the alert) is on the network. This information may alsobe used to adjust the technique discriminators used in block 545 (seeFIG. 5) and to note new attack trends. If there is a high number ofdistinct sources, an alert is added, block 845. Any new sequences foundare added to the relational database, block 840, and the process returnsto block 815. Following analysis at block 820, alert correlationproceeds to block 825, network correlation. Network correlation looksfor the same alerts on multiple sensors on the protected network. Alertsfrom related sensors are pooled into a single alert pool (logically, butnot necessarily physically).

At block 830, an examination for matches from sensor to sensor isperformed. The direction of each alert is known. An alert moving from anexternal sensor to an internal sensor should first hit the externalsystem then the internal system. When an alert on an external sensor hasa matching alert of the same source (within an appropriate time frame)on an internal sensor, it indicates the network has been penetrated(vice versa for an outgoing attack). If no matches are found, alertcorrelation ends. If matches are found, the raw packets are searched,block 835, for a reply indicating the attack worked. If so, the alertsare escalated to a high priority. Block 845 is a standard add alerts,and because the sources are known, the attacker's SIR is increased.

A global analysis may be performed including analysis of all availablesensors across multiple networks. FIG. 9 shows process flow for anembodiment of global analysis 900 according to systems and methods ofthe present invention. Global analysis is designed to analyze attacksources from the Internet across the whole customer profile. Byperforming this analysis, scanning or attack sources can be identifiedacross subnets or across the Internet. Additionally, if coordinatedattacks are occurring and are being performed against multiple sites,they can be found using global analysis. As the number of data sourcesincreases, the quality of this analysis improves.

Global analysis may be used to analyze thousands or even hundreds ofthousands of sources across the Internet at a central site in an attemptto find sources of suspicious activity. Attackers can performvulnerability scans on many targets at once, and often do so in anattempt to hide the attack. Instead of scanning one target for 500different vulnerabilities at once, they may scan 200 sites for the samevulnerabilities. The attacker's computer may be busy twenty-four hours aday, but each site receives a tiny portion of scan each day andtherefore manages to stay under the rate that is detectable bytraditional knowledge-based systems.

Referring now to the exemplary global analysis shown in FIG. 9, alertsare counted by alert type for sources outside the protected networks ofthe sensors at block 905. This yields alert frequencies by type, whichindicate attack methods being used. At block 910, alert counts are addedto the RDB. Alerts are then counted by source and priority, block 915,showing which attackers are most active on the global network. At block920, attacker information from block 915 is added to the RDB. Globalalerts are weighted by priority, block 925. Higher priority alerts aregiven additional counts in both alert type tables and attacker tables.

At block 930, in a first table, a set of alerts with a weighted countrepresenting the type of alerts on the global network is created. In asecond table, global attack sources are also weighted by priority, withboth tables being sorted from high to low. Sensors are adjusted, block935. Attacker source strengths are raised in sensor attacker databasesso alert SSVs from the same sources are automatically higher. Alert typedefault priorities may be adjusted, block 940, based on the global alertfrequencies. These can be pushed to either default priorities of alertsor to the static technique discriminators (preferred). Once globalanalysis is complete, the results are sent to each sensor for eachprotected network analyzed as part of the global analysis (see block950, FIG. 2).

Optionally, systems and methods of intrusion detection according to thepresent invention may include an alert release system to assistoperators of the IDS. Process flow for an embodiment of an alert releasesystem is shown in FIG. 10. This system is designed to reduce the numberof alerts an operator of the IDS has to view while maintaining a largernumber of alerts for correlation and analysis. Alerts are created in agenerational system, and higher generation alerts are shown on theoperator on the console. Alerts are selected for display by a series ofrules that define the alerts that should be displayed. The series ofrules are generally high level selection parameters, but may be used toselectively show or ignore alerts based on very specific criteria, suchas, for example, ignoring all alerts from a particular source orignoring all alerts in a particular time window.

The series of rules may be applied in a specific order such that once asingle rule is applied resulting in an indication that the alert shouldnot be displayed, fer rules no longer need to be applied to that alertand the alert is not displayed. Additionally, all rules may be appliedto an alert and the alert may be displayed only if a certain percentageof the rules indicated that the alert should be release to the console.For example, if ten rules are present, the system may be set up torelease an alert if seven or more rules indicate that the alert shouldbe released. Furthermore, the rules could be weighted such that theindication based on a certain (possibly, more reliable) rule is weightedheavier than the indication based on another rule. Accordingly, onlyalerts of a specific significance are displayed on the console.

The series of rules or selection rules may contain alert fields such assource, destination, source port, and destination port in addition toother critical pieces of information from behavioral analysis. SSV is astrong indicator of activity on a network. Additionally, the number ofalerts of a specific type and/or the frequency of alerts may indicatethe alert should be released.

Maintaining alerts in an intermediate storage area that is notautomatically displayed to the operator of the IDS allows additionalprocessing to be performed on alerts before they are displayed. Anothercorrelation pass could be performed to reclassify alerts not yetreleased. By looking for sequence indicators, multiple alerts may becombined into single alerts and only the single alert is displayed. Toreclassify alerts, alerts in sequence that correlate to a known type ofattack are not released, but rather a new alert is generated andreleased.

As an example of the functionality alert release system, consider theNimda worm. Five alerts are used to determine that a web attack isactually a Nimda worm attack. The five alerts are not released to theconsole, instead a new alert called Nimda is generated and this newalert is released to the console. This preserves the record that fiveattacks occurred but that they were all from the same source (as with aNimda worm attack).

An alert release system also allows sources and services to be renamed,providing name resolution for alerts so that servers (including internalservers where no public DNS is available) may be renamed. A tableresolves addresses to names. For example, before alerts are released tothe console, the addresses are changed to meaningful names, such as“firewall.” The operator of the IDS sees the name rather than theaddress, which is beneficial because the human operator is more likelyto immediately recognize the name. Address information is maintained inthe alert for the operator's use and reference.

Additionally, alerts may be renamed after alerts have been generated andimported from a signature-based server using this alert release system.Previously, all importers maintained a list of alert names that neededto changed at the time of import, each of which were unique to thesignature-based system generating the alerts. The alert release systemallows a single list of alert renames to be used to change all alerts ofa particular name to another name to assure unified data names.

Referring now to FIG. 10, where an exemplary process flow for anembodiment of an alert release system according to systems and methodsof the present invention is shown, at block 1005, alerts are added to adatabase at generation one. IP addresses are resolved to names, block1010, and alerts are renamed, block 1015. Alerts are reclassified, block1020, and a series of selection rules are used to select alerts that areto be displayed on the console, block 1025, for viewing by the operatorof the IDS. The alerts selected for display are assigned generation twostatus, block 1030, and stored in a generation two alert database.

The foregoing description of exemplary embodiments according to systemsand methods of the invention has been presented only for the purposes ofillustration and description and is not intended to be exhaustive or tolimit the invention to the precise forms disclosed. Many modificationsand variations are possible in light of the above teaching.

The embodiments were chosen and described in order to explain theprinciples of the invention and their practical application so as toenable others skilled in the art to utilize the invention and variousembodiments and with various modifications as are suited to theparticular use contemplated. Alternative embodiments will becomeapparent to those skilled in the art to which the present inventionpertains without departing from its spirit and scope.

1. A method of detecting network intrusion attempts on a communicationsnetwork, the method comprising: collecting data associated with networktraffic on the communications network, the data collection being over atime period sufficient to establish a sample of historical data, thehistorical data indicating normal network traffic; examining networktraffic for data comprising known strings and series of bytes thatindicate signature attacks; reading packets in network traffic,classifying the packets by protocols, and creating packages ofcompressed packets; applying predetermined rules to group data packetsassociated with objects on the communications network according tocommon data packet characteristics, the grouped data packetsestablishing an anomaly pool; analyzing the anomaly pool using thehistorical data; generating an alert based on the behavioral analysisand converting alerts from native signature format to a unified formatfor storage; and storing a modified version of the historical data basedon the compressed packets and the alerts.
 2. The method of claim 1further comprising: monitoring the network traffic for known strings andseries of bytes that indicate signature attacks.
 3. The method of claim1 further comprising: analyzing the anomaly pool independently of any ofthe predetermined rules that identified the data packets for addition tothe anomaly pool.
 4. The method of claim 1, wherein an alert isgenerated if a statistical value associated with the anomaly pool is notwithin a predetermined threshold value associated with the historicaldata.
 5. The method of claim 1, wherein the step of generating an alertcomprises adding alerts to an alert pool and releasing alerts to aconsole for viewing by an operator.
 6. The method of claim 1, whereinthe step of generating alerts comprises: adding a first set of alerts toan alert pool; resolving internet protocol addresses associated witheach alert in the alert pool with a name and renaming each alert in thealert pool with a name recognizable by an operator; applying a set ofrules to select alerts from the alert pool to be displayed on a consolefor viewing by the operator, the rules comprising high level selectionparameters that have been previously defined; and releasing the selectedalerts by name to the console for viewing by the operator.
 7. The methodof claim 1 further comprising: performing the method across a pluralityof networks; and compiling results in a global database.
 8. The methodof claim 7 further comprising: updating the historical data in a globaldatabase associated with the communications network.
 9. The method ofclaim 1, wherein the step of updating the historical data comprises:storing sources of possible intrusion attempts in a database with anindicator of their hostility to the network; and removing a source ofpossible intrusion attempts from the database when a value of theindicator is sufficiently low due to a period of inactivity of thesource.
 10. A non-transitory computer storage medium storing a computerprogram which, when executed by a computer-controlled apparatus, causesthe computer-controlled apparatus to perform the method of claim
 1. 11.A computer-controlled apparatus operative for implementing the method ofclaim 1, wherein the apparatus comprises a hardware server.
 12. A methodof detecting network intrusion attempts associated with network objectson a communications network, the method comprising: collecting normaldata associated with network objects on the communications network on acontinuing basis to establish historical data associated with trafficacross the network, wherein the historical data comprises deliveryinformation associated with packets communicated over the communicationsnetwork, the data collection occurring over a time period sufficient toestablish a sample of the historical data; examining network traffic fordata comprising known strings and series of bytes that indicatesignature attacks; reading packets in network traffic, classifying thepackets by protocols, and creating packages of compressed packets;analyzing the anomaly pool using the historical data; generating analert based on the behavioral analysis and a converter for convertingalerts from native signature format to a unified format for storage inat least one relational database; and storing a modified version of thehistorical data based on the compressed packets and the alerts.
 13. Amethod of detecting network intrusion attempts on a communicationsnetwork, the method comprising: collecting data associated with networktraffic, the data collection being over a time period sufficient toestablish a sample of historical data; examining network traffic fordata comprising known strings and series of bytes that indicatesignature attacks; reading packets in network traffic, classifying thepackets by protocols, and creating packages of compressed packets;applying predetermined rules to group data packets associated withobjects on the communications network according to common data packetcharacteristics, the grouped data packets establishing an anomaly pool;analyzing the anomaly pool using the historical data and known stringsand series of bytes that indicate signature attacks; generating an alertbased on the behavioral analysis and a converter for converting alertsfrom native signature format to a unified format for storage in at leastone relational database; and storing a modified version of thehistorical data based on the compressed packets and the alerts.
 14. Themethod of claim 13, wherein analyzing the anomaly pool comprises:analyzing the anomaly pool independently of any of the predeterminedrules that identified the data packets for addition to the anomaly pool;and conducting a threshold analysis to determine whether a statisticalvalue associated with the anomaly pool is within threshold values. 15.The method of claim 13, wherein generating alerts identifying possibleintrusion attempts comprises: adding a first set of alerts to an alertpool; resolving internet protocol addresses associated with each alertin the alert pool with a name and renaming each alert in the alert poolwith a name recognizable by an operator; applying a set of rules toselect alerts from the alert pool to be displayed on a console forviewing by the operator, the rules comprising high level selectionparameters that have been previously defined; and releasing the selectedalerts by name to the console for viewing by the operator.
 16. Themethod of claim 13, further comprising: performing the method across aplurality of networks; compiling results in a global database; andupdating the historical data based on results in the global database.17. The method of claim 13, wherein updating the historical datacomprises storing sources of possible intrusion attempts in a databasewith an indicator of their hostility to the network and removing asource of possible intrusion attempts from the database when a value ofthe data point indicator is sufficiently low due to a period ofinactivity of the source.
 18. A non-transitory computer storage mediumstoring a computer program which, when executed by a computer-controlledapparatus, causes the computer-controlled apparatus to perform themethod of claim
 13. 19. A computer-controlled apparatus operative forimplementing the method of claim 13, the apparatus comprising a hardwareserver.
 20. An intrusion detection system for detecting networkintrusion attempts associated with network objects on a communicationsnetwork, the system comprising: a sensor connected to the network tomonitor network traffic associated with network objects on the network,the sensor operable to collect network data over a time periodsufficient to establish a sample of historical data, the sensorcomprising: a knowledge-based component for examining network trafficfor data comprising known strings and series of bytes that indicatesignature attacks; and a packet logger for reading packets in networktraffic, classifying the packets by protocols, and creating packages ofcompressed packets; a hardware server connected to the sensor thataccepts real-time alerts for possible signature attacks and a converterfor converting alerts from native signature format to a unified formatfor storage in at least one relational database; an analysis server thatgenerates an alert based on the behavioral analysis; and at least onerelational database to store a modified version of the historical databased on the compressed packets and the alerts.
 21. The system of claim20, further comprising a plurality of sensors connected to the network.22. The system of claim 20, further comprising two or more virtualprivate network tunnels connecting the sensor to the network.